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ABSTRACT 



The introduction of training packages has focused attention 
on the quality of assessment in the Australian vocational education and 
training (VET) sector on the quality of assessment. For the process of mutual 
recognition under the Australian Recognition Framework (ARF) to work 
effectively, there needs to be confidence in assessment decisions made by 
Registered Training Organizations (RTOs) . The issues of quality and 
consistency in assessment have become central to the development of the VET 
system. Quality assurance of all aspects of assessment is crucial for the 
protection of assessors and all other stakeholders in the process. Criteria 
for quality assurance are use of qualified assessors; established validation; 
feedback; appeals and verification processes; industry-endorsed assessment 
guidelines; and provision of industry audit of the assessment process. ARF 
requirements that all RTOs must meet to obtain and maintain registration are 
to be changed substantially in 2001, reflecting the national concern with 
consistency and quality. A comprehensive approach identifies strategies to 
assure quality for all key aspects of assessment, but particularly recognizes 
the vital role of assessors and the need for their participation in 
validation activities and continued professional development. (Contains 15 
references . ) (YLB) 
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The introduction of Training Packages has focused attention in the vocational 
education and training (VET) sector on the quality of assessment. For the process of 
mutual recognition under the Australian Recognition Framework (ARF) to work 
effectively, there needs to be confidence in assessment decisions made by Registered 
Training Organisations (RTOs). The issues of quality and consistency in assessment 
have become central to the development of the VET system in Australia. However 
there are a range of views and strategies regarding how this consistency, confidence 
and quality can be achieved. 



The issue of consistency in vocational education and training 

In 1999, the Australian National Training Authority (ANTA) CEOs identified ten 
areas of inconsistency in the implementation of the national VET system by states 
and territories. These included differences with the 'declaration of vocations' and 
industrial relations arrangements for 'New Apprentices', differences in the 
monitoring and auditing of RTOs, different pricing models for user choice and 
differences with the legal underpinnings of both mutual recognition of RTOs and 
internal quality assurance arrangements for RTOs. ANTA is aiming to strengthen 
national legislation to reduce this inconsistency. 

Kaye Schofield's reviews of the traineeship systems in Queensland, Tasmania and 
Victoria (where apprenticeships were also included in the study) identified a raft of 
issues leading to poor quality provision at the system and provider level and 
recommended strategies for the registration of providers, the strengthening of the 
audit function, quality assurance of teaching and learning and specific improvements 
to the individual states' traineeship systems. In each case, Schofield identified 
assessment as one of the areas requiring attention in assuring quality. 



Consistency in competency-based assessment 

The introduction of Training Packages and increased attention to assessment have 
focused this concern with consistency. Prior to the introduction of competency-based 
training, education providers relied much more on centralised regulated systems of 
examinations to ensure consistency of assessment events. Assessment decisions were 
often made by centralised panels of 'markers'. 



Assessment against competency standards within Training Packages involves 
ollecting evidence and making judgements on whether or not the competency has 
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been achieved. Various factors including the experience and skills of assessors and 
the clarity of competency standards themselves can influence the quality of 
judgements made. The potential variability in factors affecting assessment and the 
need to maintain flexibility and focus on the learners means there will always be 
potential for variation in assessment decisions. 

In 1999, DETIR Queensland investigated the success of current assessment practices 
in supporting the new training environment (Smith 1999). The study considered the 
major issues, practices and changes impacting on the quality of the assessment of 
training and recommended ways in which assessment practices and approaches may 
be improved. 

Smith found that there is 'significant scope to improve the quality of assessment in 
terms of validity, consistency, usefulness and cost-effectiveness and that there are a 
significant number of trainers/ assessors whose practices are not supportive of a 
quality assessment system'. He found an excessive amount of assessment being 
conducted with the concept that quantity equals quality, too much emphasis on 
summative assessment, and a lack of quality educational support for 
trainers/ assessors. 

Interestingly, Schofield's report of the Tasmanian system (1999) referred to Smith's 
study. She suggested that concerns about inconsistency may be less the result of 
inconsistent practice and more the result of a competency-based training system that 
allows only for a yes/no assessment rather than graded assessment. Many trainees 
may be assessed as competent yet they will each display varying degrees of 
capability leading to perceptions of inconsistency in assessment. She concluded that 
the 'issue of inconsistency in assessment may therefore be illusory', but recognised 
that other concerns about the extent to which valid, reliable and ethical assessment 
practices are being universally applied are substantial and require attention. 

In early 2000 the National Training Framework Committee (NTFC) commissioned an 
evaluation project titled Consistency in Competency-based Assessment. The aims of 
this evaluation were to explore the concept of consistency in relation to a 
competency-based training system; identify and analyse factors that may impact 
positively or negatively on the consistency of assessment processes and outcomes, 
particularly in regard to Training Packages; and identify strategies for maximising 
consistency. The recommendations of this evaluation were concerned with the need 
to strengthen competency standards and include more specific advice in assessment 
guidelines as part of Training Packages. They also encompassed the strengthening of 
the audits of RTOs, assessor training and ongoing professional development and the 
introduction of an assessment review process for RTOs. This review process involves 
validation of assessment decisions by assessors to review the RTOs' assessment 
materials, processes and outcomes. Following this research ANT A is aiming to 
strengthen the aspects of assessment practice on which RTOs will be audited. 

The term consistency in assessment is being used here to cover the assessment 
principles of validity, reliability and fairness. For assessment processes to produce 
consistent outcomes, they must assess what they are supposed to assess and produce 
similar outcomes when used by the same assessor or different assessors with 
candidates of equal competence at different times and/or places. Assessors must 
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have a shared understanding of the standard and the basis on which assessment 
decisions are made and what constitutes sufficient evidence. 

The role of quality assurance in assessment 

Quality assurance is defined for the Training Package for Assessment and Workplace 
Training BSZ98 (ANT A 1998) as: 

A planned and systematic process of ensuring that the requirements of the 
assessment systems, competency standards and any other criteria are 
applied in a consistent manner. Quality assurance mechanisms or 
procedures are an integral part of an assessment system. 

Toop et al (1994) established a framework for an assessment system that included 
elements of a comprehensive quality assurance strategy. These included screening 
and training of assessors; verification of assessment decisions; appeals mechanisms 
and processes; and review of the assessment system. 

Bloch et al (1995), in supporting Wolf's contention that much assessment is made on 
the basis of assessor experience in their technical domain, suggested that there is a 
need to set up a system of checks and balances to justify assessment decisions. 
Quality assurance of all aspects of assessment is crucial for the protection of assessors 
and all other stakeholders in the process. 

Alexander (1997), quoted in Docking (1997, p 11), lists four criteria for quality 
assurance: the use of qualified assessors; established validation; feedback; appeals 
and verification processes; industry-endorsed assessment guidelines; and provision 
of industry audit of the assessment process. 

The international experience 

In Australia, the focus of assuring quality in assessment has concentrated on the 
'front end' of the process in developing and clarifying competency standards and 
qualifications and in the registration and audit of providers. In Australia, self- 
regulation has been the clear focus of VET policy overall. In other comparable VET 
systems such as those in the United Kingdom and New Zealand, there is a more 
centrally directed regulatory system with prescribed forms of moderation to ensure 
quality outcomes. 

The Qualifications and Curriculum Authority (QCA) in the United Kingdom 
contends that 'the credibility of any assessment system depends on fair, accurate 
assessment and effective quality assurance' (QCA 1997, p v). Effective quality 
assurance is also seen as the critical element in building the confidence of all 
stakeholders involved in National Vocational Qualifications (NVQs). 

This emphasis on quality assurance has generated a highly regulated approach to 
assessment in NVQs. Within this system, there is a dual layer of monitoring which 
makes up the quality assurance strategy - internal verification and external 
verification. The Awarding Bodies' Common Accord 1997 provides the guidelines for 
quality arrangements in the system, while Implementing the National Standards for 
Verification sets out how the national standards for assessment and verification units 
are to be assessed and verified. 



Studies by Black (1993), Eraut (1994), Lester (1996; 1997; 1999) and Konrad (1999) 
highlight some of the critical issues and concerns that they have with the highly 
regulated approach adopted for assessment of NVQs. In particular, they focus upon 
the quality control nature of the UK system which includes increasing external 
monitoring and standardisation. Lester (1996) suggests that the solution is not one of 
quality control, but rather quality assurance. He contends that the solution primarily 
rests with ongoing professional development. For him, increasing the quality control 
measures is bound to have detrimental outcomes. 

The result is likely to be that while public confidence is increased, 
validity suffers as assessment increasingly concentrates on factors which 
are amenable to checks and controls ... (Lester 1996, p 3) 

More specifically, Konrad (1999, p 9) suggests that the complex role of internal 
verifier is often under-resourced and those carrying out this role have barely 
adequate initial education and training to undertake the tasks effectively. 

Eraut (1994, p 207): 

Evidence suggests that once established by training and regular 
communication, a community of assessors is able to ensure sufficiently 
standard use of criteria; but that it is easy for standardization to slip if 
training and communication are not regularly maintained. 

The training of assessors and verifiers is another essential component of 
quality assurance, because assessment and verification are themselves 
professional processes requiring special expertise. 

The New Zealand Qualifications Authority (NZQA) adopts a somewhat less rigorous 
approach in implementing the New Zealand Qualifications Framework. 

Unit Standards are established by Standards Setting Bodies, registration of providers, 
provision of assessment materials in some industry sectors by the relevant Industry 
Training Organisation; establishment of unit standards for assessment; provision of 
assessor training and the use of qualified assessors. 

Assessments conducted against unit standards that are drawn from specific industry 
sectors are moderated through processes established by the relevant Industry 
Training Organisation. The NZQA requires providers of training to develop their 
own quality assurance systems. However, all unit standards set requirements for 
moderation, and training providers and standard-setting bodies are responsible for 
ensuring that assessors participate in the designated process for ensuring validity 
and consistency in assessment. 

Quality assurance and compliance 

In Australia, all RTOs must meet the requirements of the Australian Recognition 
Framework to obtain and maintain registration. These requirements have been 
reviewed and will be changed substantially in 2001 reflecting the national concern 
with consistency and quality. The revised ARF will stipulate requirements for: 

• systems for quality training and assessment 
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• compliance with Commonwealth and state/ territory legislation and 
regulations 

• effective financial management 

• effective administrative and records management procedures 

• recognition of qualifications issued by other RTOs 

• access and equity 

• competence of RTO staff 

• RTO assessments 

• learning and assessment strategies 

• issuing qualifications 

• use of national and state logos 

• ethical marketing and advertising. 

Regardless of any regulatory requirements, an organisation needs to be committed to 
providing the best possible training delivery and assessment services. Quality 
assurance focuses on a systematic approach to improvement that recognises and 
responds to the needs and expectations of all stakeholders in the enterprise. 

Smith (2000, p 29) concludes that: 

The consistent message from the research, worldwide, into successful 
assessment practice is that the achievement of assessment quality requires 
an integrated package of initiatives: relevant and practice-oriented pre- 
service and professional development training; a combination of both 
progressive (‘formative’) and summative assessment; a genuine ‘team’ 
effort between professional trainers and appropriate industry/employer 
representatives; on-going research into, and dissemination of, best 
practice; clearly defined standards for both initial and on-going 
‘registration’ as a trainer (as distinct from a training organisation); and an 
effective system of moderation and audit. Unless all of these components 
are addressed in an integrated fashion, quality of assessment is unlikely to 
be achieved. 



Identifying and developing quality assurance strategies 

In 2000, the Canberra Institute of Technology (CIT) and the Vocational Education 
and Assessment Centre, TAFE NSW (VEAC) investigated strategies for improving 
confidence in assessment decision making for providers (NCVER, to be published). 
Issues initially identified by providers as having an impact on the quality of 
assessment and the level of confidence of assessors and others in the VET sector 
were: 
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• the lack of consistency in assessment decisions and assessment practice 

• the new demands placed on assessors and assessment with the 
implementation of training packages 

• the lack of rigorous quality assurance processes 

• concerns about the quality of assessor training programs and ongoing 
support for assessors. 

This research drew together assessors from private and public providers specifically 
from the Children s Services, Information Technology and Horticulture industries. 
The project found that although providers used a range of quality assurance 
strategies, very few organisations currently have a comprehensive strategy to assure 
quality in assessment. From this consultation, specific strategies were identified for 
further development in the areas of validation of assessment tools and judgement, 
gathering of evidence and the use of assessment partnerships. 

Findings from this research are currently being used to develop an assessment guide 
for RTOs and assessors to develop comprehensive quality assurance arrangements 
for their organisations (DETYA, in press). The key aspects of assessment we consider 
necessary to be quality assured are: 

• the assessment system 

• the assessment process 

• the assessors 

• the evidence 

• the judgement. 

This comprehensive approach identifies strategies to assure quality for each of these 
aspects but particularly recognises the vital role of assessors and the need for them to 
participate in validation activities and continued professional development. 

Validation as a key quality assurance strategy 

Recent Australian national and state-based research points towards the need to 
support assessors in their decision making through activities to validate or moderate 
the evidence collected and the judgement made. 

The revised ARF standards (ANTA 2001, draft February) in fact stipulate that RTOs 
implement a validation process and define it as: 

reviewing, comparing and evaluating assessment processes, tools and 
evidence contributing to judgements made by a range of assessors against 
the same standards. Validation strategies may be internal processes with 
stakeholder involvement or external validations with other providers 
and/or stakeholders. 

While there are some providers and specific industry sectors currently implementing 
adhoc validation activities with assessors, there is little evidence to date of RTOs 




encouraging their assessors to systematically review, compare and evaluate their 
assessment processes, tools and evidence (NCVER, to be published). 

(Note that the use of the term 'moderation' is being debated. Moderation has 
traditionally implied some statistical application to achieve norm-referenced scores 
such as the moderation activities for the NSW Higher School Certificate. In the case 
of competency-based assessment, the term validation is now used to describe the 
process of reviewing, comparing and evaluating assessment.) 

Implementing these QA strategies - where to from here? 

Tension between national compliance requirements and implementing quality 
assurance strategies by RTOs 

The revisions and strengthening of the ARF Standards for RTOs are being made in 
an attempt to improve consistency and quality of VET provision. These changes 
could become a whitewash to ultimate improvement of quality if RTOs treat them as 
maximum requirements rather than minimum standards. The RTO has to sponsor or 
own this process and have the will to devote resources to improving the quality of its 
training and assessment. 

Difficulties in implementation of quality assurance strategies 
The implementation of validation processes will certainly require RTOs to allocate 
additional resources for assessors to meet to compare and evaluate their assessment 
effectively. Our experience to date in working with RTOs suggests that while this 
process offers excellent opportunities for professional development and improving 
quality of assessment, providers will find it difficult to allocate the resources needed 
for effective operation. 

Tension in emphasis in implementation of the range of quality assurance strategies 
This paper has referred to the range of different strategies for assuring quality in 
assessment and stressed the need to implement all of them comprehensively. 

It will be interesting to observe developments in attention to these strategies for 
assessment of VET courses in secondary schools. The issue of consistency of 
outcomes has become particularly significant with the inclusion of VET subjects 
linked to the ARF being used for university selection and is of great concern to 
ANTA CEOs. In a paper commissioned by ANT A, the Assessment Research Centre 
(Griffin et al 2001) recommends a range of strategies to deal with this complex issue, 
where VET subjects can contribute to what has been a single university entrance 
score for Australian universities. 

The paper recommends the introduction of a standards referenced system for 
interpreting and reporting of VET subjects, with content-specific criteria developed 
for each vocational subject. Greater specificity in industry standards would be 
welcomed nationally. 

The paper also recommends the development of calibration of assessment data 
through item response modelling procedures, which can be calibrated onto a single 
underlying differentiating scale for each VET subject or unit. Griffin et al (2001) 
specifically recommend that 'all VET assessment data that combines workplace and 




classroom assessments to integrate competency and differentiating purposes should 
be centrally calibrated to enable quality assurance mechanisms to apply 7 (2001, 
Recommendation 11). A system is suggested where schools could design and 
administer performance tasks within approved assessment strategy and design 
parameters. The data from these assessments could then be provided to a central 
administrative agency for calibration. However, such measures would be costly, 
difficult to administer and potentially lack the flexibility needed to enable assessment 
that demonstrates competence in authentic workplace environments. 

While the paper does identify a need for professional development for teachers in 
assessment task design, it focuses on the development of a national task bank. There 
is consideration of the potential role of moderation to support assessors. In this 
context, it is seen as the process of bringing scores and standards into alignment and 
the emphasis on the calibration of assessment tasks to ensure comparability across 
schools. 

The task of balancing the needs of assessment for industry competency recognition 
and university selection is extremely complex. However, it will be interesting to 
watch the development of these recommendations and their potential viability in the 
broader VET context. 

Conclusion 

We have defined confidence in assessment decision making as a sense of trust in the: 

• quality of the information provided to learners 

• validity of the assessment tools 

• reliability and fairness of the assessment procedures 

• sufficiency of evidence 

• correctness of the interpretation 

• accurateness of the recording and reporting of assessment results. 

These factors must be included in the design of an RTO system to assure the quality 
of assessment. Minimum standards related to these factors are embodied in the ARF 
standards for RTOs. However there is a danger that they remain minimum 
standards, as some of the most valuable features such as activities for validating 
assessment will be costly to implement. In the effort to emphasise consistency rather 
than confidence, there may also be undue emphasis placed on strategies to 
standardise processes rather than provide resources to support assessors to perform 
their work with more confidence. 

It is interesting to consider whether confidence in assessment decision making 
requires consistency of approach. The concept of consistency may ultimately be 
illusory as Schofield has suggested, but the confidence of providers, assessors and 
learners in assessment decision making certainly needs to be enhanced. 
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